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Introduction 


The commercial game, Portal 2, is a puzzle-based game in which the user solves 
puzzles presented in a series of “test chambers.” Upon the successful completion of a 
puzzle the user is allowed to exit that particular chamber and thus makes progress 
within the game. The associated level design tool, The Puzzle Maker, enables users to 
create additional rooms (test chambers) for the game. 


The Broken Rooms lesson was designed by Geoff Moore and Lisa Castaneda so 
students would manipulate pre-built rooms, however, the rooms had inherent design 
errors and were therefore unplayable as presented. Essentially, the rooms were 
erroneous examples of how a real room would work. Students would need to identify 
and then fix the rooms at the design level so they would become playable. The task of 
fixing the rooms would be too simple for students if they were allowed to use any tool 
available in their solutions. Therefore, additional constraints were imposed upon each 
broken room that limited the options students had for fixing the room. It was expected 
that students would find it much more difficult to fix a room if there were constraints 
involved and thus the cognitive demand of the lesson would be higher. The link for the 
lesson is included in the reference section. 


This paper is an exploration of a few general questions the author had about using The 
Broken Rooms with middle school students. Would the use of erroneous examples 
challenge the students in a different way than simple level-design problems? Would 
student knowledge from fixing the broken rooms enable them to effectively create their 
own broken rooms? Does differential skill-level within the game affect the problem- 
solving skills required in the lesson? Research on erroneous example use in math 
classrooms, particularly through the use of video games, is still limited. It is hoped that 
the observations from this exploratory teacher research will be beneficial to future work 
in this area. 


Theoretical Framework 


The theoretical framework draws from research on the relevance of gaming in the 
classroom. It extends the idea of playing games to that of designing games and, in 
addition, incorporating erroneous examples into the process. The foundations of 
problem solving in mathematics are explored within the context of student work with 
errors. It is suggested that fixing errors requires a different set of skills than simply 
identifying errors in the first place. Previous work with the same group of students in a 
series of level-design lessons demonstrated that although creating levels utilizes a 
variety of skills, additional problem-solving skills might be utilized in a framework that 
incorporated erroneous examples. 


The Common Core Standards for mathematics highlight the importance of persevering 
in problem solving and monitoring progress in order to change course if a solution is not 
working (2013). The ability to identify faulty reasoning while monitoring one’s own 
progress is a valuable life-skill and demonstrates a higher-level of thinking and 
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reasoning than following procedural steps when problem solving. The lesson was 
designed to draw out these reasoning and reflective skills. 


The use of video games in the classroom has received more attention from educators in 
the last ten years and in fact, some investigators argue that video games provide an 
excellent model for best practices in teaching (Willis, 2011; Gee, 2007; McGonigal, 
2011; Prensky, 2006). Although educators might hesitate to use commercial games in 
the classroom, there are commercial games which may have applications within an 
academic setting even when not designed to do so. Valve software’s Portal 2 game 
and its related level-design tool, The Puzzle Maker, are examples of commercial games 
with academic relevance. The immediate nature of feedback when using video games 
as a learning tool is a crucial benefit. Chang et al. (2012) found that the speed and 
nature of feedback provided by video games encouraged students to complete and 
engage in problem solving tasks. The iterative nature of problem solving in The Broken 
Rooms was enhanced through the design and playability of the tools. 


Although designing game levels with The Puzzle Maker is a creative process and 
involves problem posing and solving it was thought that perhaps including a framework 
which incorporated erroneous examples might result in a deeper level of thinking on the 
students’ part. Tsovaltzi et al. (2009) describe erroneous examples as, “worked 
solutions including one or more errors that the student is asked to detect, explain, 
and/or correct.” (p 1) In the case of The Puzzle Maker, the worked solutions were pre- 
designed rooms that were not playable because they had an error fundamental in the 
design. This exploration of anomalies is theorized to be a catalyst for helping students 
to gain a deeper, philosophic understanding of a topic (Egan, 2010). The intent was to 
provide the students with a scenario where their assumptions would be challenged and 
their reasoning skills put to use. 


Polya (1945) suggested that mathematical problem solving occurs in four phases: (1) 
understand (2) plan (3) carry out and (4) look back. The iterative nature of work with 
game design fits well within this four phase framework and the sequence was familiar to 
the middle school students from their previous work in math class. In fact, the second 
portion of The Broken Rooms takes this problem-solving framework a step further by 
employing Leung’s (1993) model of the four phases: pose-problems, plan, carry out and 
look back. With the essential change being in the first phase as students are creating 
problems for others to solve. 


In the understanding phase, from Polya’s framework, students needed to identify the 
error inherent in the room. Tsovaltzi et al. (2009) observed that for middle-level 
students (sixth grade) there was a dissociation between correctly identifying an error 
and knowing how to fix it. In traditional math classes, identifying an error in a structured 
math problem can be challenging for middle school students, but the steps often follow 
a clear pattern or are algorithmic in nature. However, when the problem is more 
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abstract, such as when using a level-design error, it might be even more challenging to 
identify the error because of the open-ended nature of the problem. Tsovaltzi 
referenced this idea when suggesting that students often follow well-practiced steps in 
traditional math problems, making much of the work procedural. By moving the 
problem-space to something far-less procedural it was theorized that students would 
need to access higher order thinking skills since simple recall or declarative knowledge 
would not be helpful in this space. 


The planning phase would require that students examine the constraints imposed upon 
the room and find solutions that might fit within those limitations. The carry out phase 
would require students to make the requisite changes to the room design. Students 
would complete the /ook back phase once they had successfully rebuilt and played the 
room, reflecting on whether or not the solution fixed the problem effectively. This four- 
phase problem solving method, regularly used in the math-classroom, seemed like a 
relevant model to use when working with the problems in The Broken Rooms. In the 
second portion, where students designed their own broken rooms, Polya’s phase one 
was simply replaced by Leung’s so that students were instead creating the problem to 
be solved. 


It has been suggested that by focusing on errors in a classroom setting students are 
internalizing faulty information (Borasi, 1994). However, in courses like computer 
science and programming, it is expected that students will note and correct their errors 
through processes like debugging (Melis, 2004). Graham (2004) suggests that open- 
source software actually has fewer bugs inherent in it because the very nature of open- 
source material recognizes that bugs might exist and allows flexibility in addressing 
them. This debugging approach to problem-solving mirrors much of what students were 
asked to do within this lesson. Approaching errors as a natural part of the learning 
process helps to remove the stigma associated with mistakes. 


Borasi (1994) noted that using errors as springboards for mathematical inquiry was a 
stretch for mathematics courses back in the mid-nineties; many educators were not 
interested in using inquiry in this way. However, several positive results of using errors 
directly in the instructional process were noted in her study. One of the most salient for 
this paper is the democratic culture the discussion of errors brings about in an 
educational setting. As students questioned their thinking, they pushed their instructor 
to reconsider her own thinking and relevant discussions ensued. Borasi describes this 
interaction as, “sending a powerful message to the students about the value of errors.” 
(p 180) It was hoped that this kind of mutually beneficial student-teacher environment 
would occur within the context of this lesson and help students to see that errors are an 
essential part of the learning process. 


Creating rooms requires a basic level of understanding and knowledge in order to 
design a chamber that functions effectively within the constraints of the game. In 
addition, research conducted by Ohlsson suggests that correcting an error inherent in 
the room enables the player to improve their future actions and decision-making as the 
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process itself encourages them to identify and remove faults in their general underlying 
knowledge (Ohlsson, 1996). Examining the rooms, trying to locate the error, eliminating 
extraneous information, considering constraints imposed in the problem-space and then 
fixing the room so that is playable requires several steps from the students. As they are 
completing these various tasks, their general understanding of the rules and functional 
components of the game continue to evolve. 


It was theorized by the authors that prior knowledge of the game was crucial to the 
success of this lesson. Ohlsson (1996) highlights the importance of prior-knowledge 
when examining error detection by suggesting that prior knowledge allows students to 
discard information that is extraneous and enables them to monitor their own 
performance. It was predicted that students’ experience with the erroneous examples 
might differ based on prior skills and knowledge. In particular, students with more 
experience in the game and this type of problem-solving would be able to think more 
flexibly about the errors and possible solutions than those students who had a smaller 
knowledge base. 


Methods 


The lesson was introduced after students had multiple exposures to both the game and 
the level-building tool. The lesson took three to four class periods spread out over a 
three week period. The lesson was introduced to sixth, seventh, and eighth grade 
students, in their respective math classes, at a private suburban grade school. There 
were a total of 36 students; 16 sixth graders, 10 seventh graders and 10 eighth graders. 


The protocol suggested in the attached lesson was followed. The instructor began by 
leading a class discussion about the purpose behind the game and of individual test 
chamber rooms to ensure that all students had a common understanding of the 
necessity of solving a test chamber in order to make progress in the game. It was 
reiterated that the point of the test chamber was to progress from the entrance to the 
exit so that the character could leave the chamber. 


The instructor then engaged students in a discussion concerning which components 
made a level engaging and challenging to play. Students were asked to give a variety 
of examples and they listed things such as multiple tasks per chamber, accurately 
placing portals, getting correct timing, and figuring out the proper sequence of tasks 
associated with a given room. 


At that point, the concept of a broken room was introduced along with the suggestion 
that there may be multiple solutions available to fix a given room. The introductory 
room was shown as an example to the entire class. Students were all asked to rebuild 
the room shown on their own laptops and to find a solution that fit within the constraints. 
The next page contains a screenshot of the introductory room along with its listed 
constraints: 
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Constraints — 


Not allowed to: change portable 


surfaces or add anything from the 
asset panel 


Introductory screenshot for class discussion with related constraints 


All students were given several minutes to solve the sample room. When the majority 
of the class had found a solution, students took turns sharing. Students were 
encouraged to share ideas that did not work along with explanations for why they did 
not work. The modeling of both plausible and non-plausible solutions was an important 
part of the discussion of the iterative nature inherent in the lesson. The instructor then 
ensured that students had a clear understanding of what a broken room was through 
further questions and comments. Once it was clear that students understood the 
concept, the student packets were distributed. 


Students were given two screenshots of a series of five pre-built rooms that they would 
need to replicate on their own laptops. In addition to the two screenshots of each room 
provided to students, additional screenshots were included at the end of the teacher 
resources in case students needed more support in reconstructing the rooms. This was 
not the ideal way to share the structure of the rooms. Unfortunately, due to limitations in 
the platform, students were unable to access rooms that other people had pre-built 
easily, thus the instructor could not pre-build the rooms and have students download the 
pre-constructed levels. Students were encouraged to look at all five rooms in the 
packet and determine which rooms looked most appealing to begin their work. The 
rooms were placed in the packet in no particular order of difficulty. Once students found 
a room that was intriguing, they used the screenshots to rebuild the room. Accuracy of 
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rebuilds was monitored as one of the authors of the lesson was also the instructor in 
this case and was able to provide corrections in room replication if needed. This was 
essential because if students had errors in the replication of the room they could be 
solving a very different problem than the one intended by the authors. 


After students rebuilt their selected room the accuracy of the construction was verified 
by the instructor before students were allowed to proceed. At this point, students were 
asked to identify the error present in the room. In some cases error identification 
occurred immediately upon screenshot examination, in others students located the error 
as they constructed the room and still others needed to try to play the room as it was to 
identify the problem area. Once the error was identified, students were asked to reflect 
on the constraints associated with that particular room. Constraints were listed below 
the screenshots in the student packet. Students were then allowed as much time as 
needed to play around with the room and experiment with the various devices shown on 
the asset panel in their efforts to find a solution. 


When a student felt a possible solution to the room was reached, they were asked to 
find a peer for whom they could demonstrate their proposed solution. The peer was to 
confirm whether or not the given solution met the constraints. Often students were quite 
excited by their solutions and asked to share with the instructor or multiple peers. 


After students had completed a minimum of two broken rooms they were then asked to 
work on the final chamber problem. The final chamber was an open-ended problem. 
Students were asked to create a room, ensure it was broken in some way, take two 
screenshots of the room and write up a list of constraints themselves. They were then 
asked to find a peer who was willing to try to complete their room. A list of reflection 
questions was also answered at this point in the process. 


Observations and Discussion Test Chambers 1-5 


Since this work took place in an active classroom environment within the context of 
regular Pre-Algebra and Algebra 1 classes, observations and example data were simply 
noted by the instructor. As a teacher/researcher, other tasks also had to be attended to 
during the observation periods (management, technical issues, interruptions from 
visitors, etc.). 


It was hoped that all students would have sufficient time to complete all five pre-built 
broken rooms prior to constructing their own. The amount of time needed for all 
students to rebuild and then fix the rooms was underestimated by the instructor. The 
process would have gone significantly faster if students did not have to reconstruct each 
room. Although from a spatial reasoning perspective, there was value in having 
students try to reconstruct the room. In fact, student difficulties with reconstructing the 
room and their abilities to rotate and reflect components of the room served as the 
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impetus for an additional lesson constructed by the lesson authors more focused on 
spatial reasoning skills. 


Students with more experience using the game, regardless of whether they were sixth, 
seventh, or eighth graders, had an easier time rebuilding the rooms and rebuilt them 
with less errors than their counterparts. This rebuilding component impacted the flow of 
the lesson. The more experienced Portal 2 players and experienced gamers were better 
able to build the rooms quickly and had more time to try to solve multiple rooms. 
Therefore, the students who had the most exposure to the largest variety of broken 
rooms were also the students who could most quickly reconstruct the rooms. However, 
this challenge aside, once a given student had a room to work with, interesting 
differences in approaches to the problems could be found. 


Once students had reconstructed a room they were asked to identify the error that 
existed within that room. Students with the most experience with the game were often 
able to identify the error in the room simply by the screenshot. Two of the most 
advanced players, a sixth grade boy and eighth grade boy, expressed frustration that 
they even had to build out the room to “fix” it because they felt they could clearly 
articulate the problem and solution with no work. Those students were encouraged to 
try to find a more interesting solution to the problem than the one initially identified in 
order to have a more challenging experience. 


The vast majority of students found that the error in the room became apparent as they 
reconstructed the room. Comments such as, “Hey, wait, there is no way to access that 
door” or “Oh, | see, there is no possible way for you to reach that ledge,” were heard 
repeatedly as students built. These comments were often met by other students with 
frustration because they had not yet had a chance to fix that particular room and they 
felt those comments were taking away from their experience of the assignment. 
Interestingly, the fact that students were allowed to progress through the packet in any 
order was helpful in that regard. It was not easy for students to cheat by looking at their 
neighbor because very often their neighbor was doing something else entirely. 


The final minority of students were typically those with the least experience in the game 
or very often female students who were not gamers. These students either could not 
identify the error through the building process and needed to play the game in order to 
find it, or, they weren’t sure they had correctly identified the error and wanted to play it 
to make sure. Comments from this group of students were of this type: “I don’t think this 
room is broken,” or, “I think you made a mistake, Mrs. C, this seems to work.” 
Occasionally the room did work for them because there was an error in their 
reconstruction that slipped by the instructor. At other times, the students were in-game 
and seemed to misconstrue what was really happening as a result of their own actions. 
For instance, a female eighth grade student felt that there was no error in Test Chamber 
2 and attributed her lack of progress to the fact that she could not properly place the 
initial portals in order to make it to the first ledge. In other words, she attributed her 
struggles to her lack of skill, not inherent design flaws in the room. This type of 
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comment was interesting. Her feelings of inadequacy in the playing of the game 
actually caused her to question whether or not the room was broken even though the 
entire lesson was based on the premise that each room absolutely was broken and 
needed to be fixed. 


Frustration on the part of the students was extremely common once they had identified 
the error in the room and then reflected on the error in light of the constraints imposed 
upon their solution. The phrase, “This is impossible. There is no way to solve it with 
those constraints!” was repeatedly heard across all grade levels. In most cases, this 
just seemed like an initial gut-response as students recognized the challenge of what 
was ahead of them. In other cases, students really seemed to internalize this idea and 
it became a sort of block to any forward progress in solving the puzzle. One strategy 
employed by the instructor to try to quell student concerns included the assurance that 
not only was there was one possible solution, but multiple ways address it. 


Often when stuck students were encouraged to more thoroughly examine the 
constraints in light of available options they became more willing to explore and engage. 
For instance, in Test Chamber 1 (below) most students easily recognized that the level 
was broken because there was no feasible way to reach the exit. There were a long list 
of components which could not be used in order to fix the room (devices, gels, panels, 
switches, faith plates, or cubes). When students would say there was no way to fix the 
room, the instructor would ask them to open the asset panel and look at all available 
tools. When they were encouraged to look item by item and compare items on the 
panel to those that were listed on the constraints one at a time, they would discover that 
there were several items on the asset panel whose use was not restricted. 


Screenshot of Test Chamber 1; Exit is the open round door at the top-left corner 
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Test Chamber 1 was not a particularly difficult one to solve. However, the range of 
solutions students derived was really quite staggering. Although some were quite basic 
(using an aerial faith plate to jump up) other students came up with very inventive 
solutions (multiple tractor beams, a series of light beams). Their motivations for trying 
these different solutions varied widely. Several students indicated they picked their 
solutions because they were simple and would allow them to quickly move on toa 
different room. Others had already figured out a simple solution, or saw another student 
use a simple solution and purposefully wanted to challenge themselves to build a more 
complex solution. Interestingly, it was often the least-experienced players that came up 
with the most complex solutions to the problems, not because they were trying to do 
something inventive but because they did not have a solid understanding of the 
mechanics of the game. An eighth grade student remarked after her complicated build, 
“Oh, geez. | totally didn’t even realize you could do something more simple! | could 
have been done, like, 20 minutes ago.” 


The sixth graders in particular seemed to thrive on the finding of unique solutions that 
differentiated themselves from their peers. The seventh and eighth graders seemed 
more interested in finding any possible solution just so they could move on, even if the 
solution was not unique. 


Lower-achieving students were more apt to give up, get frustrated and switch rooms 
without coming up with a satisfactory solution. Higher-achieving students were often 
more interested in trying to find creative ways to solve the problem. However, higher- 
achieving students were also more eager to find loopholes by which they could bypass 
the intended routes of solution-finding. 


For instance, in Test Chamber 2 (next page) in all classes, someone quickly figured out 
that one could bypass all of the constraints simply by lowering the wall of fizzlers in the 
center of the room. This was a challenging room for all levels and our hope was that 
students would try a more complex solution set that involved changing some flip-panels 
in the room. Very few students wanted to try to find a more interesting solution to this 
room. For advanced students this lack of alternate solution-finding seemed to be due to 
the theoretical limits on what they could actually do due to the constraints. For novice 
students the lack of alternate solution finding seemed to stem from mechanical 
problems they had in actually navigating the working portions of the room. For instance, 
several seventh and eighth graders asked for instructor help to even make it to the first 
ledge so they could test their ideas (they would consistently fall into the toxic goo). 
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Test Chamber 2; Entrance and exit are directly across from each other divided by a wall 
of fizzlers. 


The room selected the least by all levels of students was Test Chamber 3. Students 
clearly expressed that because it involved graphing on an x, y, z axis they were reticent 
to try it. Even though this was a math class, students seemed less eager to approach it 
because they would actually have to use coordinate graphing to solve it. Even though 
the option to solve it with two coordinates was available (and this option was pointed out 
by the instructor, especially to the sixth graders) very few students wanted to have the 
task of only working within a given set of coordinates and then in addition, trying to fix 
the errors. Even among the more advanced students who completed all five of the 
rooms, this room was consistently ranked as their least-favorite room to solve (next 


page). 
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Zaxis 


Y axis 


P xan 


Test Chamber 3 


Overall, feedback on the first five test chambers was extremely positive. The lesson 
was designed to fit within a once-a-week video game activity the instructor had 
previously set-up earlier in the year. Students would repeatedly ask for opportunities to 
work on solving the broken levels even on non-scheduled game days. 


When asked why they enjoyed the problems nearly all students agreed that it was very 
fun to try to figure out what the lesson authors had done wrong in making the rooms. 
Students felt that because there were multiple solutions they could be more creative 
and had a better chance of finding a way that worked. Several commented that it made 
their brains “work a different way.” A few of the less-skilled players also said they felt 
less pressured because they weren’t designing a room from scratch, they were simply 
trying to find a way to fix a problem within the room and that was less daunting than 
creating a full-room. 


Female students were more likely to collaborate than male students. This was a pattern 
the author observed in several previous instances of using video games in the 
classroom. The female students seemed to enjoy having a partner to bounce ideas off- 
of and several who did pair-up even went on to find different solutions than their 
partners; they just seemed to feel more comfortable having a peer to support their 
thinking. 


From an instructional perspective, having students come up with a solution and then 
justifying their thinking to another student forced them to better articulate their 
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reasoning. The peer check-in both served to push students to clearly explain why their 
solution fit within the constraints or to then confront the fact that their reasoning was 
incorrect. On several occasions, students took these opportunities to discuss and share 
the differences in reasoning and approaches to solving a particular room. In more than 
one instance, particularly in sixth grade, the instructor was called upon to determine if a 
given solution violated a constraint when a pair was arguing. 


Students also expressed enjoyment in the fact that the instructor was often impressed 
by their creative and varied solutions. Several times, in all classes, the instructor 
commented, “We didn’t even think of that when we wrote the problem!” Particularly in 
room two, students were intrigued to hear what the actual “answer” was (recall that this 
was the room where most students found a very simple solution despite the authors’ 
intentions). Even though their answer was not exactly what the instructor was hoping 
for, there was considerable glee at all grade levels that they had found a way to “work 
around” the constraints and find a simple solution. In seventh grade, one girl remarked, 
“Hey, you guys, that’s cheating.” A great discussion followed considering which 
constraints were listed and if a “fizzler” counted as a wall (which students unanimously 
agreed it did not) and therefore it was not a violation of the stated rules. Being able to 
argue with the instructor and defend their thinking often resulted in cheers and obvious 
enthusiasm on the part of the students. 


The overall positive comments, engagement, on-task behavior, and related discussion 
during these activities was very high. The flexibility to pick rooms (and switch) if they 
were stuck seemed to help students stay engaged. Oddly, the fact that they had to 
rebuild the rooms sometimes seemed to serve a valuable purpose of giving them a 
break from the concentrated problem solving to something very concrete they could 
manipulate and make steady progress with. 


Observations and Discussion-- Student Generated Chamber and Reflection 


After successfully completing a minimum of two broken rooms over the course of two 
class periods, students were allowed to begin work on the “Final Chamber.” This 
chamber required that they construct their own broken room, create a list of constraints 
and then have a peer attempt to fix the room. They were then instructed to complete a 
list of reflection questions. 


For some of the more novice students, the barrier to entry in this portion of the lesson 
became apparent. They felt the challenge of not only creating a level but then 
somehow breaking it was significantly harder than simply correcting someone else’s 
error. For afew students the prospect of this was overwhelming. Their lack of a sense 
of efficacy in effectively building rooms was amplified by the fact that the rooms had to 
be broken. This sense of hesitation speaks to the effectiveness of the previous work 
with the broken rooms, however, because it was apparent to many students at this point 
that creating a broken room involved more than just making a room that was playable 
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and removing items from it to break it. In fact, this led to a larger group discussion in 
both sixth and seventh grade about what made the broken rooms work. Students 
decided amongst themselves that building a basic room might be a place to start, but 
that simply removing items might result in rooms that were too simple. 


Several students pointed out, at all grade levels, that just because they might remove 
something from a room, such as a tractor beam, did not necessarily mean that the 
player would try to put that item back. The sixth graders actually discussed whether or 
not it mattered. Did the player have to put the room back exactly as the designer 
intended? The general consensus was no, as long as the room met the constraints. 
However, these discussions demonstrated that students were thinking about the 
interaction between their intentions in the room design and the intersection point 
between that intention and the choices made within the context of the constraints. 


As expected, this portion of the lesson was time consuming and challenging. However, 
the open-ended nature of the problem enabled students to work within their skill levels. 
Different patterns emerged based on skill level in the complexity of their room designs, 

challenges they faced, and feedback they gave. 


Very advanced students designed very advanced rooms, but also often assumed a very 
advanced skill level on the player’s part. The instructor often had to provide corrective 
feedback to these students on the complexity of their rooms. The levels often had 
multiple rooms attached together and the sheer complexity and types of errors and 
constraints they imposed made finding solutions very challenging. There needed to be 
enough structure inherent in the design and a high-level understanding on the player’s 
part to know where things could be fixed and manipulated within the room. This was 
frustrating for the advanced designers, and, often they were annoyed when players did 
not clearly see what needed to be fixed. The designers attributed any challenges 
students had with the reasoning to be due to a lack of skill on their peers’ part rather 
than an inherent flaw in their own design thinking. 


The attitudes amongst the more advanced players were almost condescending. These 
players had a very clear image of how the room should be played and did not seem as 
comfortable when peers inadvertently deviated from the designer’s intended plan. At 
one point, when the instructor mentioned that she was not sure exactly what was wrong 
with a portion of a room the student sighed and said, “Clearly, in this portion of the room 
you need to be able to get to the hidden ledge in the upper portion of the screen so that 
you can enter the black hallway and you can’t because there is no mechanism for you 
to access the hidden ledge.” When the instructor found a different solution for reaching 
the ledge the student expressed concern that the instructor’s solution was not as good 
as his intended one, even though both met the constraints. 


The higher skill level players’ efficacy with the game served to negatively impact their 
use of the logical thinking inherent in the playability of a broken room. These players 
wanted to create flashy rooms with many problems. Although they clearly understood 
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how to fix the broken rooms they had solved previously, and they immensely enjoyed 
solving their peers’ rooms, they lost sight of the purpose of creating an inherently 
solvable problem within the broken rooms framework. The complexity of their rooms 
interfered with their understanding of how erroneous examples might work best. 


In contrast, the less experienced players often heavily criticized themselves for the 
simplicity of their rooms. One eighth grade student even said, “I’m not that good at 
Portal. My broken room is lame.” Even though the sense of self-efficacy in this group 
was lower, their rooms often made a great deal more logical sense in terms of how the 
design actually fit with the constraints they imposed than those of the very advanced 
students. It was clear that these students relied heavily on the strategy of building a 
basic room and them removing an item from it. Their constraints were more vaguely 
defined and reflected a lower-level of understanding of the possibilities within the game. 


These students also seemed to express a bit of frustration when peers solved their 
rooms with a variety of methods. Their interpretation of multiple solutions seemed to 
prove, in their minds, that their room was too simple and any solution would work. Their 
lower level of understanding of the game impacted their ability to manipulate both the 
tools of the game and to see the larger construct of how the erroneous examples could 
best be used to create an interesting problem. 


The students who seemed to have the best grasp of how the erroneous examples 
worked within the context of a broken room were the middle-level kids. They were 
comfortable enough with the game itself to easily manipulate the room, they had 
enough exposure to a variety of broken rooms that they understood the premise well, 
and they had enough background knowledge of the tools of the game to formulate ideas 
about how peers might attempt to solve the rooms. These students seemed to derive 
satisfaction from the fact that their peers had a number of strategies for solving their 
rooms. Comments indicative of this group were, “Oh! Beth just solved it in a totally 
different way than | intended. | didn’t think of that!” These students would seek the 
instructor’s feedback and would often comment that they had a vision for their room but 
wanted to make sure the constraints guided their peers in the right direction. This group 
used more of an iterative process and would often stop, go back, and change elements 
of their room based on peer feedback or their own thought processes. 


One of the more interesting elements of this portion of the lesson was the reflection 
question that asked students to think about the difference between “a solution” and “the 
most efficient solution.” In retrospect, this would have been an excellent discussion to 
have after students played the rooms the instructor designed but prior to making their 
own room. However, as it stood, students still had in-depth discussions and arguments 
at all grade levels. Regardless of in-game skill level most students had opinions about 
what this meant along with concrete examples from their own game play and that of 
their peers. This also led to discussions at all levels of a “solution” versus a “good 
solution.” Some students were perfectly happy with a completely inefficient solution 
because they couldn’t come up with anything else. Other students felt that finding the 
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most efficient way to execute the solution and correct the error was the point of the 
exercise. There was a contingent of students who really valued finding creative, out-of- 
the box solutions to problems and others who felt that if it wasn’t efficient, then it was a 
waste of time and energy and was not as “good.” These discussions in all three classes 
concluded with an acknowledgment that these types of problems were open-ended and 
the ultimate goal was to simply fix the error. One of the more advanced sixth grade 
boys referenced the fact that if he were playing the game for real he would look for the 
most efficient solution but in this lesson, he didn’t have to do that and he could be more 
creative in his solutions, so he was. 


Conclusion 
Observations 


Through the use of The Broken Rooms, the author was able to begin to better explore 
how erroneous examples can be utilized in the classroom through video games. 
Students did struggle with the complexity of the erroneous examples and related 
constraints, but most were able to overcome this frustration in order to successfully 
solve the required rooms. As anticipated, designing erroneous examples was more 
challenging for students than simply correcting the rooms and correcting the rooms was 
more difficult than simply identifying the error. Skill level in-game had interesting 
impacts on problem-solving ability, particularly in the high and low-level students. 


Video games in the classroom provide ample opportunities for repeated practice, an 
important step in the learning process (Gee, 2007), in an environment that is engaging 
and responsive. The student feedback on the broken levels was extremely positive and 
students had many opportunities to problem-solve and enhance their knowledge base 
throughout the course of the lesson. Students seemed to enjoy finding the mistakes 
and coming up with unique ways to fix the rooms. Creating the rooms was a more 
challenging task, across all skill levels, but valuable in the sense that it required a 
deeper level of understanding of the game and the logic behind the broken rooms. The 
goal of the lesson was to get students to use higher order thinking skills such as logic, 
reasoning, metacognition, critical thinking, and problem solving in a unique setting that 
allowed some flexibility, but provided enough structure, that there were ways to 
measure the functionality of their solutions. The lesson, as designed, largely did this. 
Students were heard verbalizing their thinking, reflecting out-loud with one another 
about the effectiveness of various solutions, and articulating the various iterations of 
their process. 


These conversations were different in nature than those that occur when students 
simply create levels. In a previous series of lessons, primarily focused on free-form 
level design, student comments centered more on the logistics of placing items in the 
room and making it fun for peers to solve. During The Broken Rooms, student 
comments were much more focused on problem-solving and reasoning through the 
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various scenarios. There was depth to the conversations and students seemed curious 
to hear more about why their peers selected one particular solution over another. There 
were inquisitive discussions about why one student perceived the problem a particular 
way or whether or not a solution really worked well within the constraints. Many 
students commented at the conclusion of the lesson on the overall difficulty of fixing the 
erroneous solutions. One seventh grader remarked, “It would have been way easier if 
you just let us do whatever we wanted to fix the rooms.” 


It is also interesting to note that some students, in this case often the lower achieving 
students, came up with solutions that would meet the constraints but perhaps were not 
the most efficient or effective. In many cases, their proposed solutions would fit the 
constraints but didn’t always align with the real-world play of the game. This finding was 
similar to one noticed by researchers at the University of Washington on the protein 
folding game, Fold-/t. Since not all Fo/d-/t players have the requisite knowledge of how 
protein folding works in the “real world,” they could create folds that met the 
requirements of the game, but from a scientific perspective would never actually occur 
(personal communication, July 2013). Similarly, in the Broken Rooms, students could 
sometimes derive an idea that would work in the game if the player was by some 
chance thinking about the room exactly as the designer perceived it. However, 
sometimes student designers did not see the larger picture of how the erroneous 
example fit within the scheme of the game and the variety of ways their peers might 
approach the problems. These designers missed seeing the larger context of the 
problem solving and limited themselves to a specific scenario in-game. 


Although there was no computational math involved in this lesson, it was an opportunity 
for students to extend their reasoning and problem solving skills in a more open, 
creative format. Articulating their thinking, collaborating with peers, negotiating their 
way through the constraints and trying to create errors in their own reasoning were all 
valuable elements of a holistic problem-solving process. The problem solving 
framework in this lesson closely mirrors the types of problem solving seen in middle 
school-level math classes: understand/pose-problems, plan, carry out, and look back 
(Polya, 1945; Leung, 1993). It was helpful to demonstrate that this sort of process is 
applicable even outside of traditional math problems. In fact, the nature of the game 
absolutely required the look back phase, a step students are often less eager to 
complete in a traditionally structured lesson. 


Work with students on erroneous examples is not only engaging, but requires them to 
demonstrate a much broader understanding than simply showing they can regurgitate a 
specific skill-set. Designing rooms that function demonstrates one level of 
understanding. It shows that students have a certain grasp of the fundamental 
elements of game design and can effectively create a problem for a player to solve. 
Being able to use design-thinking to look at the fundamental room structure and identify 
errors in thinking demonstrates a different type of understanding. Students need to infer 
what the designer was thinking, use their understanding of the game and their problem 
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solving skills to find a solution within a fixed set of constraints and then test that theory 
by playing the level. Finally, being able to design their own room with an error and 
constraints upon the player requires the designer to fundamentally know the game, 
forces them to think about how others might solve problems, think logically enough that 
they can break their own design and then design constraints that will guide the player 
into thinking in a similar way. This is a huge cognitive demand on a middle school level 
student. 


Often American mathematical education is focused on behaviorist principles which try to 
extinguish mistakes from the problem-solving process (Radatz, 1979). There is much to 
be learned from one’s own mistakes as well as in the process of identifying mistakes in 
the reasoning of others. Worked examples have been used successfully to help 
students develop skills in the problem-solving process by enabling them to fill in 
procedural steps; in addition, erroneous examples provide the opportunity for students 
to further extend that learning. Students must draw upon a firm knowledge base in 
order to have the requisite background to enable them to find the error. From there, 
they have to think critically about possible solutions, articulate the error and take action 
in order to find the solution (Ohlsson, 1996). As students worked to fix the erroneous 
examples, it appeared that they were also addressing fundamental flaws within their 
own understanding about the way problems could be solved in-game. 


The ability to create and then play is one of the great advantages to using video games 
in the classroom. The ease of use of the Puzzle Maker tool, ability to save and rebuild 
and to play each other’s creations with limited hassle is a huge advantage. This 
combination of solving problems, discussing them, creating them, and then analyzing 
the results in a semi-formalized way is an evident advantage of using technology such 
as video games. 


When students would suggest that a room wasn’t really broken or that perhaps 
something was impossible, they could quickly manipulate the various elements in the 
room to test it out. If something didn’t work as planned, with a few keystrokes they 
could step out of first-person mode, fix it and go back in to try again. The iterative 
nature of video games, along with the immediate feedback and ease of use of this level- 
design tool, is of huge benefit. In these circumstances, students could focus on testing 
their solutions and finding the errors in real-time. In addition, the conversations, 
arguments and group discussions enabled a variety of reflective opportunities, which is 
where the learning is really brought to the surface level for analysis and connections 
were formed in student understanding. 


Using Bloom’s Taxonomy can provide a helpful model when looking at the types of 
learning that might be taking place when utilizing an erroneous examples lesson such 
as the Broken Rooms. Simply rebuilding the room requires only knowledge level 
thinking (the first level on the taxonomy). Solving an erroneous example requires the 
additional elements of knowledge, application and analysis. Students are required to 
understand the problem before them and extrapolate from the constraints imposed. 
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They then need to apply their abstract knowledge within the concrete setting supplied. 
As they explain their reasoning to others they are reaching the level of analysis as they 
articulate their thinking and ensure it does indeed apply the relevant pieces of 
information. 


The students’ creation of their own broken level layers in the synthesis and evaluation 
levels of the taxonomy. In this case, as Sherry and Pacheco (2006) describe, students 
are taking ideas and reassembling them into a new whole. Once they have the concept 
of a broken level they are asked to combine the requisite elements representing this 
synthesis portion. Finally, through the construction of the final chamber students are 
asked to evaluate their rooms and the rooms of their peers to see if there is consensus 
upon what constitutes a solid broken room. This lesson is scaffolded so that it can 
guide students into higher levels of content processing. 


Much like Ohlsson (1996) described, it appeared as though there were two distinct 
competencies required of students: the ability to declare the error and then the 
subsequent ability to correct it. In addition to those two skills, the inclusion of a third, 
the ability to create an error, seemed to push students to think more broadly about their 
problem solving. It gave them the opportunity to explore the problem solving space 
from a variety of different perspectives resulting in a more holistic experience. 


The observations in this paper were made while the instructor was also maintaining a 
regular classroom. Obvious challenges arose in documenting student comments and 
dealing with technical difficulties in addition to the other interruptions that are 
commonplace in a traditional classroom setting. However, there is value in actually 
examining a particular type of instruction in a real-world setting versus a laboratory 
setting. It would be helpful in future work to either record student responses or to have 
more than one adult present in the room so that additional observations could be noted. 
It would also be intriguing to look at differences in terms of understanding with a group 
of students who simply construct levels versus students who use the erroneous 
examples to learn and solve problems. 


It would also be informative to use a more detailed evaluative framework for the 
solutions in future work, similar to one employed by Chang et al. (2012). Within that 
framework, students could more easily reflect on the flexibility, accuracy, and originality 
of the solutions. Leung (1997) also developed a cognitive analysis scheme to more 
accurately assess problem-posing scenarios. Being able to talk about the solutions 
using specific measures would have value in terms of helping students to think about 
what made “good” solutions versus solutions that just happened to work. 


In future endeavors, it would be beneficial to spend time after students have worked 
with the erroneous examples to have them articulate what makes erroneous examples 
useful as a learning tool. Taking the time to reflect upon the purpose of the examples 
may help to clarify additional misconceptions. Student understanding of how to solve an 
erroneous example is very different than their understanding of how to create one. 
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Creating their own requires that they not only have a solid understanding of how the 
game itself works, but what types of thinking a player might have as they approach a 
problem and what would make an interesting but logical erroneous example. The final 
chamber design would have likely gone more smoothly for students of all levels if the 
instructor had spent structured time going back over the various erroneous examples 
with the class and clearly discussed how the solutions fit the rooms, what the instructor 
had hoped for in terms of outcomes versus the actual outcomes, and brainstormed with 
students some new ideas of erroneous examples. 


It would also be helpful to design a more rigorous study of the use of erroneous 
examples versus worked problem or perhaps self-posed problem scenarios. Although it 
was evident to the instructor from the student comments, questions and work products 
that the use of erroneous examples was cognitively demanding of students, this was not 
a controlled study and there were no measures in place to quantitatively show this was 
the case. Given the limited research on the use of erroneous examples as learning 
tools, particularly in video games, additional work in this area could be very informative. 


Indeed, many positive attributes came forth as a result of this lesson. Students were 
highly motivated and enthusiastic, in fact, for the entire three weeks, the instructor 
would regularly be asked if it was “Broken Rooms” day because the students were 
eager to work on the levels. The instructor found errors in her own thinking and was 
able to model these corrective adjustments for the students, resulting in a more 
democratic classroom model. In addition, students were eager to point out the 
instructor’s errors and a general sense that it was to be expected that errors would 
occur permeated the classrooms. Student skills within the game construct and their 
understanding how to work within an imposed constraints framework improved with 
exposure and practice. The multiplicity of solutions in The Broken Rooms allowed for 
recognition, on both the students’ and the instructor’s part that a variety of solutions 
were possible. Finally, through the use of erroneous examples it appeared that students 
were thinking more deeply and in more complex ways than through traditional problem- 
solving means. Additional work with video games and erroneous examples will 
hopefully support these findings and perhaps help to demonstrate the importance of this 
type of problem-solving within the classroom setting. It is hoped that these qualitative 
observations might highlight the value of continued work in this area. 
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